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ABSTRACT 

Three 3-day assessment workshops were h«ld in 
Boulder, Colorado from June 19-29, for personnel in the assessment 
field from state departments of education. Seventy-six participants 
from 35 states, Puerto Rico, the Virgin Islands and the District of 
Columbia attended. Two of the three workshops concentrated on 
National Assessment as one model for large-scale assessments. 
Presentations were made by National staff members. The third worksnop 
was held between the other two. The program for this workshop 
concentrated on seven different state assessments, a short up-datina 
on National Assessment, and small group discussion sessions. 
Presentations were made by state directors of assessment, contractors 
representatives, and selected National Assessment staff. Evaluations 
of the workshops were quite positive, in all three workshops over 90 
percent of the responses indicated a desire to attend a follow-up 
workshop in 1974. Host questions elicited very favorable reactions, 
-he maior suggestions for improvement were for more discussion tim«» 
and better speakers. The things liked best were the opportunity to 
interact with other assessment persons and to hear about other 
programs. (Author/DEP) 
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ASSESSMENT WORKSHOPS 
ABSTRACT 

Three 3-day assessment workshops were held in Boulder, Colorado from 
June 19-29, for personnel in the assessment field from state departments of 
education. Seventy-six participants from 35 states, Puerto Rico, the Virgin 
Islands and the District of Columbia attended. 

Two of the three workshops concentrated on National Assessment as one 
model for large-scale assessments. These were held at the beginning and the end 
of the nine days. These workshops were for personnel who had not attended a 
similar workshop in either 1971 or 1972. Presentations were made by National 
Assessment staff members. Attendance was 25 and 17. 

The third workshop was held between the other two. Attendees were personnel 
who had attended a National Assessment workshop before, as well as any of the 
attendees at the other two workshops who cared to stay. The program for this work- 
shop concentrated on seven different state assessments, a short up-dating on National 
Assessment, and small group discussion sessions. Presentations were made by state 
directors of assessment, contractors representatives, and selected National Assess- 
ment staff. Attendance was 53. 

Evaluations of the workshops were quite positive. In all three workshops over 
90 percent of the responses indicated a desire to attend a follow-up workshop in 1974. 
Most questions elicited very favorable reactions. The major suggestions for improve- 
ment were for more discussion time and better speakers. The things liked best were 
the opportunity to interact with other assessment persons and to hear about other 
programs. 
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INTRODUCTION 



This third series of asstssment workshops was decided on almost 
at the conclusion of the 1972 workshops. It was obvious then that state 
directors of assessment and'their staffs were anxious to continue the 
brief but well-established tradition of the workshops. A six-month 
follow-up confirmed that decision. 

Initial plans called for two workshops, one to be devoted to 
National Assessment and one to state assessments. A mid-winter survey 
of--interest demonstrated that the number of "new'* participants would 
be too large to handle effectively in a single workshop. A decision was 
made to offer two identical Natiori3l Assessment workshops, one before 
and one after the state assessment workshop for previous workshop 
participants. 

Program development was related as much as possible to the 
evaluations received from attendees in 1972, within the usual constraints 
of staffing and time. A slight shift in local locale was necessary because 
of crowded conference conditions at the University of Colorado. 
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WORKSHOP PURPOSES 



In the last two years the trend to state assessments has moved 
from A position of "interest" to a position of "implementation. " At 
least a third of the states have gathered assessment data in one form or 
another and others are very close to that point. Almost all states ai-e 
engaged in some phase of planning for or implementing an assessment. 

In this climate it is imperative that information about large-scale 
assessments should be shared between and among the various states and 
the National Assessment project. A workshop to bring together the 
directors of assessment and members of their staff is a logical vehicle. 
National Assessment, as a nationwide project, is in a position of natural 
leadership to handle this need. 

Previous workshops had concentrated on National Assessment as 
one model of assessment. For persons who had attended one of those 
workshops it was more logical to plan a program that would emphasize 
planning and activities within several states that are implementing assess- 
ment in a variety of ways. Any cursory examination of state plans shows 
on^; that, while no two are identical, there are certain common elements 
such as "use of released National Assessment exercises" by some, or 
••use of a regression model of analysis of results" by some, or "problems 
of reporting to lay publics" by others. Thus, one purpose of the 1973 
workshops was to provide a sharing of state assessmv.nt plans. 

A second major purpose of the workshops was to continue the 
activities of the previous two years, an indepth review of National Assess- 
ment policies and procedures, for interested state department personnel 
who had not attended a previous workshop. This was felt to be important 
for two reasons: (1) some states had not sent anyone to a previous work- 
shop, and (2) many states had had changes in and/or additions to their 
assessment staff. 

Clearly we are past the stage of deciding whether to do state 
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assessments or not. We are in a rapidly expanding period of implemen- 
tation* The ultimate success of any of these programs, as well as the 
National Assessment project, will depend upon the quality of the program 
and the relevance of the results, A sharing of information among the 
states and National Assessment certainly is one essential ingredient for 
ultimate success. 
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WORKSHOP PLAN 



The basic ingredients for program planning were: (1) previous experience 
in 1971 and 1972; (2) six-month evaluations of workshop participants in 1972; (3) a 
pre-registration of participants; and (4) knowledge of existing state assessment 
plans. In addition, the work of the Cooperative Accountability Project in contracting 
for monographs on dissemination of assessment results and developing large-scale 
assessments contributed to the planning. 

Rather large pre-registrations for the National Assessment Model workshop 
led to a decision to split the groiip into two (hereafter called Workshop I and 
Workshop HI), Identical programs were planned for the two groups (persons who 
had not participated in the similar workshops in 1971 or 1972). These workshops 
were condensed into 3-1/2 and 3 days respectively, by reducing the time devoted 
to certain topics, by eliminating specijac discussions of state assessments, * and 
by utilizing evening sessions. While attendees did not divide themselves equally 
between Workshops I and IH, the division did permit greater flexibility of discussions. 



Attendance was 25 and 17 respectively. 




1973 

Workshop Schedule 




June 19 20 21 22 23 24 25 26 


27 28 29 30 


weekend 

Workshop I X X X X 


Workshop n XX 


X 


Workshop III 


X X X X 



*Attendees interested in specific state assessments were invited to attend 
Wli'kshop II also, 
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Workshop n was planned to concentrate primarily on state assessment 
programs^ except for one-half day devoted to "Wliat*s New'*- in National Assessment. 
One full day was scheduled for presentation and discussion of four topic areas of 
assessment^ as each area is being handled in four different states — Calfomia, 
Florida, Maryland, and Texas. Another day was planned to discuss how three 
different states used contracting agencies to handle a specific phase of their assess- 
ment program. The state -contractor combinations were Minnesota-Research 
Triangle Institute; New Jersey-Educational Testing Service; and Mchigan-CTB/ 
McGraw-Hill, A third day was planned in small group discussion sessions on topics 
chosen by conference participants, and individual consultation with National Assessment 
staff. Attendance at Workshop 11 was 53 (including 18 persons who also attended 
lor m). 

Three outside speakers were engaged to discuss the general topic of dissemi- 
nation of educational information. Drs. Gerald Miller and Erwin Bettinghaus, 
Professors of Communication at Michigan State University, spoke at Workshops I 
and in respectively. Dr. Louis Rubin, Professor at Large and Communications 
Specialist, University of Illinois, spoke at a conference dinner during Workshop n. 

^ ^ife 

The location of the workshop was at the Royal Inn, adjacent to the University 
of Colorado campus in Bouldar, Colorado. Conference arrangements were made 
through the University, but meeting rooms were more convenient at the Inn. Some 
participants were housed in a University dormitory; others in the Royal and Holiday 
Inns. The University Club was used for the conference dinner. 
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June 19. 
1:00 - 3:00 
3:00 . 4:00 



4:00 - 5:30 

June 20 
9:00 - 10:00a.m. 
10:00 - 10:30 
10:30 - 12:00 

12:00 - l:30p.m^ 

1:30 . 3:00 

3:00 . 3:30 

3:30 . 4:30 

7:00 - 9:00 

June 21 

9:00 - 10:00a.m. 

10:00 . 10:30 

10:30 - 12:00 

12:00 - 1:30 p.m. 

1:30 . 3:00 

3:00 . 3:30 

3:30 . 4:30 
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WORKSHOP I SCHEDULE 
June 19-22, 1973 

Conference Registration 

Introductory Remarks: Chairman Irv Lehmann 

Welcome: James Hazlett, Administrative Director, 
NAEP 

Why Assess?: Chairman Frank Womer 

NAEP Slide Show: Jack Schmidt, State Assessment 

Developing Goals and Objectives: Hal Wilson 

Developing NAEP Objectives: Scott Newcomb 
Break 

Developing Assessment Questions, Items, Exer- 
cises: Hal Wilson 

Lunch 

NAEP Exercise Development: Nancy Burton 
Break 

Principles of Sampling: Don Searls 
NAEP Sampling: Todd Rogers 

Discussion of Sampling: Don Searls, Todd Rogers 
Break 

NAEP Data Collection and Scoring: Roger Talle, 
Sue Oldefendt 

Lunch 

NAEP Data Analyses: Sue Sherman 
Break 

NAEP Data Analysis and Report Preparation: 
Charlie Gadway 
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June 22 

9:00 - 10:30a.m. Discussion of Data Analysis and Report Prepara- 
tion: Rex Brown 

10:00 - 10:30 Break 

10:30 - 12:00 Principles of Dissemination 

Introduction: Tom Fisher, Michigan Department 
of Education 

Presentation: "Dissemination of Assessment 

Information'* - Gerald R. Miller, 
Professor, Michigan State University 

(this presentation sponsored jointly by NAEP/ 
Cooperative Accountability Project) 

12:00 - 1:30 p.m. Lunch 

1:30 - 3:30 NAEP Utilization/Applications/Interpretation: 

Ken Seaman and U/A staff 

Discussion of Utilization/Applications/lnterpretation 

Services Available from Utilization/Applications: 
Jack Schmidt and U/A staff 

3:30 . 4:00 Break 

4:00 - 4:30 Windup and Evaluation: Womer and Lehmann 
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WORKSHOP II SCHEDULE 
June 25-27, 1973 



June 25 

Directors of Assessment from four different states, California, Texas, 
Florida and Maryland discussed alternative approaches to assessment 
in purposes and materials development, data collection, analysis and 
dissemination, 

8:00 - 9:00 a.m. Conference Registration 

9:00 - 10:30 Purposes, Objectives, Exercise Development 

Chairman: Frank Womer 

Participants: 

California: Dale Carlson, Lorrie SV .pard 

Florida: Jim Impara 

Maryland: Jim Fisher, Roberta Keiter 

Texas: Keith Cruse 

10:30 . 11:00 Break 

11:00 - 12:00 Sampling and Data Collection 

Chairman: Irv Lehmann 

Participants: Dale Carlson 

Lorrie Shepard 
Jim Impara 
Jim Fisher 
Roberta Keiter 
Keith Cruse 

12:00 - 1:30 Lunch 

1:30 - 2:45 Analysis and Report Writing 

Chairman: Irv Lehmann 

Participants: Dale Carlson 
Jim Impara 
Lorrie Shepard 
Jim Fisher 
Roberta Keiter 
Keith Cruse 

2:45 - 3:15 Break 
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3:15 - 



4:30 



Interpretation /Dissemination/utilization/ 
Applications 

Chairman: Frank Womer 



Participants: Dale Carlson 

Lorrie Shepard 
Jim Impara 
Jim Fisher 
Roberta Keiter 
Keith Cruse 

6:00 - 9:00 National Assessment Dinner 

Chairman: Ken Seaman, Director of *^ ^ 

Utilization /Appli cations , NAEP^^ 

Speaker: Louis Rubin, Communications %/ 
Specialist, University of Illinois 

**Dis semination of Educational Perform- 
ance Information** ^ 

June 26 

9:00 - 10:30 a.m. What's New at NAEP? 

(Objectives, Exercise Development, Sampling, 
Data Collection): George Johnson, Associate 

Staff Director, NAEP 

10:30 - 11:00 Break 

11:00 - 12:00 What's New at NAEP? 

(Analysis Reporting, Dissemination): 
George Johnson 

12:00 - 1:30 p.m. Lunch 

Afternoon Sessions: A team from each of three states discussed one 
aspect of assessment in that state. Each team consisted of a State 
Director of Assessment and a representative from a contracting agency 
that worked with that state. 

1:30 - 2:45 Development of a State Assessment Plan in 

Minnesota 

Chairman: Ftank Womer 

Participants: John Adams, Minnesota 

Dave Bayless, Research Triangl 
Institute 

John Pyecha, Research Triangle 

^ Institute 
18 8 



2:45 - 3:15 Break 

3:15 - 4:30 Development of Objectives and Exercises in New 

Jersey- 
Chairman: Irv Lehmann 

Participants: Gordon Ascher, New Jersey 
William Schabacker, ETS 

7:00 - 9:00 Development and Tryouts of Exercises in Michigan 

Chairman: Jack Schmidt, NAEP 

Participants: Dan Schooley, Michigan 

Fred Finch, CTB/McGraw-Hill 

June 27 

9:00 - 10:00a, m. Small Group Discussions on topics selected by 

conference participants 

10:00 - 10:30 Break 

10:30 - 11:30 Repeat of Small Group Discussions 

11:30 - 12:00 Windup and Evaluation: Irv Lehmann 

(Optional) 

2:30 - 4:00 Individual consultation at NAEP offices in Denver 

with NAEP staff 
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WORKSHOP III SCHEDULE 
June 27-29, 1973 



June 21 



8:00 - 9:00a.m. Conference Registration 

9:00 - 10:00 Introductory Remarks: Chairman Frank Womer 

Welcome: George Johnson, Associate Staff 
Director, NAEP 

Why Assess ?: Chairman Frank Womer 

NAEP Slide Show: Jack Schmidt, 

10:00 - 10:30 Break 

« 

10:30 - 11:30 Development of Goals and Objectives: Hal Wilson 

11:30 - 1:00 p.m. Lunch 

1:00 - 2:30 Development of NAEP Objectives: Scott Newcomb 

2:30 - 3:00 Break 

3:00 - 4:30 Development of Assessment Questions, Items, 

Exercises: Hal Wilson 

7:30 - 9:00 NAEP Exercise Development: Nancy Burton 
June 28 

8:30 - 9:30a.m. Principles of Sampling: Todd Rogers 

9:30 - 10;00 Break 

10:00 . 12:00 NAEP Sampling: Todd Rogers 

12:00 - 1:30 p.m. Lunch 

1:15 - 2:45 NAEP Data Collection and Scoring: Roger Talle, 

Wayne Martin 

2:45 - 3:00 Coffee 

3:00 - 4:30 Principles of Data Analysis and Technical Report 

Preparations: Ina Mullis, Dave Wright 

4:30 - 5:30 NAEP Data Analysis and Report Preparation: 

Rex Brown 
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June 29 



9:00 - 10:30 a.m. Principles of Dissemination 

Introduction: Dan Schooley, Michigan Department 
of Education 

Presentation: "Dissemination of Assessment 

Information*' - Erwin P. Bettinghaus, 
Professor, Michigan State University 

(this presentation sponsored jointly by NAEP/ 
Cooperative Accountability Project) 

10:30 - 11:00 Break 

11:00 - 12:00 Discussion of Principles of Dissemination: 

Bettinghaus 

12:00 - 1:30p.m. Lunch 

1:30 - 3:30 NAEP Utilization/Applications Interpretation 

Jack Schmidt and U/A staff 

3:30 - 4:00 Coffee 

4:00 - 4:30 Windup and Evaluation: Womer and Lehmann 
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EVALUATION OF WORKSHOP EXPERIENCES 
WORKSHOPS I AND m 

On the final day of the workshop, the participants were asked to 
complete an evaluation form. The purpose of this evaluation was two-fold: 
(1) to provide the workshop staff with some objective evidence concerning 
the impact of the workshop experience on the participants, and (2) to pro- 
vide the workshop staff with feedback that could be of value in the planning 
and conduct of future workshops. 

The major questions asked in the evaluation form for Workshops I 
and III were as follows. The evaluation of Workshop II is covered separately 
in the following section of this report. 

1. Relative value of major presentation sessions. 

2. General impressions of the workshop experience. 

3. Length of workshops. 

4. Suggested improvements for future workshops. 

5. Most and least valuable workshop experience. 

6. Degree of satisfaction with workshop experience. 

7. Suggested improvements. 

Twenty-seven participants were registered for Workshop I. Of these, 
two only attended one or two sessions and therefore were not considered in 
determining the population size. Of the 25 eligible respondents, completed 
evaluation forms were received from 24 participants (96 percent). The 
results are based on the replies received from these 24 participants. 
Seventeen participants were registered for Workshop III. Of these, one 
participant attended about one-half the workshop and two attended only one 
day. The former was included but not the latter. Responses were received 
from all the participants. 

In a few instances, the total number of responses to a particular 
question will be less than the base number of participants because of failure 
to respond to that question by one or more persons. 

12 
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Relative Value of Major Presentation Sessions 



One of the more difficult aspects in the planning of any learning 
experience is to offer experiences that will be most beneficial to a majority 
of the Itarners, Planning a workshop is no different. On the basis of 
evaluations conducted with previous workshop participants, the directors 
planned and organized a series of speakers and topics within the constraints 
that the general topic was National Assessment and that most speakers 
would be from the NAEP staff. 

The participants were asked to rank, in order of importance, the 
eight major presentation sessions in relation to their present and possible 
future work. Using a weighting scheme where 1 = the activity ranked first, 
the mean ratings and rank order are presented below. 





Mean Rating 


Rank 


Order 


Session 


WS I 


WS m 


WS I 


WS m 


Developing objectives 


2.5 


3.4 


1 


2 


Sampling 


4.5 


2.0 


5 


1 


Scoring 


5. 1 


5.7 


8 


8 


Report preparation 


4.9 


4.9 


7 


4 


Exercise development 


3.4 


5. 1 


2 


5 


Data collection 


4.8 


5.2 


6 


6 


Data analysis 


4.0 


5.4 


3. 5 


7 


Dissemination 


4.0 


4. 1 


3.5 


3 



It is evident from the results that there are both similarities and 
differences between the participants in the two workshops in their ratings of 
the various content sessions. For example, participants in both workshops 
rated developing objectives and dissemination in the top three; and rated 
scoring of least value. In contrast, whereas the Workshop III participants 
felt that the sampling session was of most value to them, this was not true 
for V/orkshop I participants, who ranked sampling fifth. Workshop I par- 
ticipants ranked exercise development second, whereas it was fifth for 
Workshop III. 
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Although there may be marked differences in the rank order of a 
particular session (for example, report preparation) there is little or no 
difference in the mean rating between some of them. This is an artifact 
of the ranking procedure and suggests that one should look both at the mean 
rating and rank order before drawing any conclusions, especially where 
there is a marked discrepancy in rank order. 

There is no denying the fact that there is a marked difference 
between the participants in the two workshops insofar as the sampling and 
the exercise development are concerned. This may be due, in part, to 
the differences between the participants in terms of training, experience, 
and job responsibilities. Also it might be explained in part by differences 
in the speakers and format employed in the two workshops. Whereas Work- 
shop I used a team approach for sampling, only a single presenter was used 
in Workshop III. On the other hand, speakers were identical for exercise 
development in both workshops. 

It may be of interest to note that two of the participants did not answer 
the question, but said "all topics are of great importance in the planning and 
implementation of an assessment program. " Informal comments could lead 
one to feel that this sentiment was shared by other participants, even though 
they answered the question. 

Another point of interest is the greater homogeneity of responses for 
Workshop I. The mean ratings went from 2. 5 to 5. 1, a difference of only 
2. 6 ranks. For Workshop III, however, the mean ratings went from 2. 0 
to 5. 7, a difference of 3. 7 ranks. This may have been due to a leveling 
effect introduced by a larger group of raters, or the interests within Work- 
shop I may have been more heterogeneous (resulting in less differentiation 
when rankings were averaged). 

In Workshop I, developing objectives and exercise development were 
clearly of greatest intent, whereas no one topic was clearly of least interest. 
In Workshop III sampling and developing objectives were of greatest interest and 
seoring of least interest. 

14 




General Impressions of Workshop Experience 

The reactions from the two workshops are so divergent that it does 
not seem appropriate to discuss them collectively. Rather, the data are 
presented separately, with an attempt to interpret the variability in results. 
The data from this question are presented below. 

Workshop I Workshop III 



Area 


Good 


Average 


Poor 


Good Av 


erage 


Poor 


Overall workshop organization 


14 


8 


2 


8 


7 


0 


Workshop content 


11 


10 


2 


9 


6 


0 


Speakers /discussion leaders 


9 


13 


1 


9 


6 


0 


Workshop materials 


17 


7 


0 


10 


5 


0 


Living facilities 


20 


2 


1 


9 


3 


2 


Opportunity for group discussion 


9 


8 


7 


12 


3 


0 


Opportunity for individual 
interaction 


11 


10 


3 


12 


3 


0 


Availability of workshop co- 
directors 


17 


6 


0 


13 


2 


0 



The major differences between Workshops I and III were in terms of: 
(1) the time frame (Workshop I was conducted in 3-1/2 days; Workshop III 
in 3 days); (2) some presenters; and (3) the size of the groups. 

The most positive reactions (positive in the sense that 70 percent or 
more of the participants rated the question **good**) for the Workshop I 
participants were living facilities (83%), availability of co-directors (74%), 
and workshop materials (71%). In contrast, "goodness** reactions for the 
Workshop III participants. were availability of co-directors (87%), opportunity 
for discussion (80%), opportunity for individual interaction (80%). Thus, 
three of the eight questions suggested considerable satisfaction in each 
workshop. However, only one question availability of co-directors 
received this level of satisfaction from the participants of both workshops. 
Slightly more than one-half of the participants 587o and 53% in Workshops 
I and III respectively rated overall workshop organization as *'good. ** 
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The Workshop I p«-*.rticipant8 were more negative in Iheir reactions 
in comparison to their counterparts in Workshop III. For example, the 
onl/ question that received any ''poor" ratings from Workshop III partici- 
pants was living facilities (two out of fourteen, 14 percent). 

For Workshop I six of the questions elicited at least one "poor'' 
response, out of 24 respondents. But for only two of these six questions 
was the percentage greater than 10 percent of the respondents (3 or more). 
For "opportunity for individual interaction'' there were 12 percent ''poor" 
ratings, and for "opportunity for group discussion" there were 29 percent 
"poor" ratings. This is very understandable when one considers the size 
of the two workshops: Workshop I at one time had 27 participants whereas 
Workshop III at one time had a maximum of 17 participants. Also, the 
fact that the participants were housed in different locations (more of the 
Workshop III people lived in the same place), might explain some of the 
findings. 

It is interesting to note, however, that of a total of 16 "poor" 
responses from Workshop I, 9 of them came from three respondents. 
This suggests that Workshop I may not have met the needs of three of the 
participants too well. 

The matter of group dynamics must also be considered when attempting 
to interpret the markedly different results to the question on opportunity 
for group discussion. The memory of the directors suggests that roughly 
comparable amounts of time were spent in group discussion in Workshop I 
and Workshop III. But it is possible that a more limited number of the 
participants in Workshop ! actually engaged in discussion and raised ques- 
tions, and that others, perhaps not by choice, were precluded from doing 
so because a few used up the available time. Or, it is possible that the 
sheer size of the group and the tight scheduling prevented everyone from 
being able to participate in the discussion. These reactions suggest that 
the size of future introductory workshops be seriously considered by the 
staff. 
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The question on quality of speakers/discussion leaders received 
more ''average" ratings than "good" ratings in Workshop I. Certainly the 
speakers at the workshops differed markedly in their teaching ability and 
if one had to select a single reaction that was voiced most frequently, it 
centered about the quality of the speakers. Specific comments were made 
by six Workshop I participants and by three Workshop III participants. It is 
difficult to control this problem completely, because not all researchers 
are automatically good "teachers. " Certainly efforts should be made to 
exercise some greater control in the future. 

Comments made by more than one participant in Workshop I were 
as follows: 

"More careful culling of speakers. . . some are very knowledgeable 
but poor speakers" (6) 

"Better coordination among speakers" (3) 
"Co-directors were attentive to the group's needs" (2) 
"Too recture-oriented--not a workshop" (2) 

"NAEP people tended to speak to each other rather than answer the 
participants* questions" (3) 

The other comments, made by only a single participant, were con- 
cerned with such things as physical facilities, the technical nature of the 
content, living facilities, and the like. 

The only comment made by more than one person in Workshop III 
was related to the quality of the speakers used (3). The other comments, 
made only once, dealt again with physical facilities, evening sessions, 
and the technical nature of some of the material. 

In summary, it would appear that the majority of the participants in 
both Workshops I and III had positive reactions and felt that most of the 
experiences offered would be or are of value to them. The staff are con- 
cerned about the average and negative comments related to the speakers 
and opportunity for discussion and will take them into considera^on in 
future planning. 
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Length of Workshop 

On the basis of remarks made at the workshops in 1971 and 1972, 
and because of time constraints for some of the NAEP staff who were 
serving as speakers and resource personnel, it was decided to reduce 
the length of the workshops from five to three and one -half days and to 
three days. This was done without omitting any of the essentials about 
National Assessment covered in previous workshops but by removing 
one day on state assessments. This did result in long days and some 
evening sessions. The participants were asked for their reaction to this 
condensation and the results were as follows: 

1. Nineteen participants in Workshop I and 10 participants in 
Workshop HE stated it was satisfactory; 

2. Two participants in Workshop HE said that the material 
should have been spread over more days; and, 

3. Four participants in Workshop I and one participant in Work- 
shop III said that less material should be covered in the 
same number of days. 

Initially, the co-directors were concerned that the participants would 
be "worked** too much considering the length of the day sessions and the 
evening sessions. It would appear that the majority of the participants 
76 percent and 77 percent in Workshops I and III respectively felt that 
they were not being pushed. However, some of the comments made on the 
evaluation sheet such as '*too scheduled-missed opportunity to interact, " 
**avoid night meetings, "although much material was covered in a short 
time, to reduce the amount of time would be undesirable and to increase 
the time too much would be impractical, " suggest that some consideration 
be given to the question of workshop length another year. If it would be 
considered desirous to permit more group interaction and group discussion, 
it would appear that the length of the workshop would need to be increased 
or the enrollment limited to about 15 to 20. 
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Four of the Workshop I and six of the Workshop HI participants made 
specific comments. Three of the former and one of the latter group said 
that the sessions were too long and that evening sessions following a very 
full day are almost too much. One Workshop III participant said that 
"some of tlie material was repetitious. . . some could be condensed further. . . 
emphasis could be shifted. 

Most and Least Valuable Workshop Experience 

As would be expected with a diversity in the training, experience, 
and job responsibilities of the workshop participants, those experiences 
most valuable to some would be less valuable to others. In fact, it would 
appear that a common frame of reference is the nature of one's job respon- 
sibility. As some participants stated, ''sessions on , because the 

concerns discussed were more directly related to my present tasks. " 

The most common comments made as "most valuable" experiences 
were that the workshop provided them with an opportunity: (1) to meet 
with others who had similar problems; (2) to hear about National Assess- 
ment and thereby develop a better understanding of that model and see 
how it could fit (either directly, partially, or completely) into their own 
state assessment plans; and (3) to discuss points of mutual concern. 
Possibly the following comment succinctly portrays the feeling expressed 
by a majority of the participants: "The real strength of this first week in 
my opinion is the quality of informed personnel whom I have had the good 
fortune to meet and whom I consequently will pursue to provide profes- 
sional assistance as necessary. " This feeling is one that has been present 
in all the workshops and it is evident that further contacts are being made 
between the various state assessment personnel attending the workshop. 
On more than one occasion the directors have heard about participants 
calling each other to discuss problems or to offer suggestions. 
Specific comments that were made were as follows: 

"Discussion of the development of objectives and exercises. 
These two activities are necessary in any assessment pro- 
gram" (6) 
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'^Meeting NAEP staff and the opportunity to interact with 
people from other states*^ (6) 

^'Familiarization with NAEP'' (5) 

"Lea -^ing about pitfalls one falls into in an assessment pro- 
gram ana alternative actions" (4) 

"Sampling and dissemination sessions were excellent" (4) 
"Knowing what^s going on in other SDE's" (4) 
"Learning about NAEP sampling techniques" (3) 
"Principles of dissemination by Miller" (3) 

Two other comments accurately portray the consensus of feeling of 
the majority of the participants regardless of which workshop they attended. 
They are; 

"Sharing of ideas among participants. Each one now has 
the opportunity to improve what he has been doing in his 
particular state. " 

"It has enabled me to make a much more accurate assess- 
ment of the potential value and feasibility of adapting parts 

of the NAEP model for state assessment in ^and the 

care with which such decisions must be approached. The 
opportunity to exchange ideas and information and the under- 
standing of NAEP (which was the objective) were the most 
valuable aspects of the workshop, " 

It is interesting to note that over 20 percent of the Workshop I 
participants commented that the most valuable experience was learning 
about NAEP whereas none of the Workshop III participants mentioned this. 

❖ * 

The least liked portion of the program by the participants in both 
workshops were the sessions on report preparation (10 persons). However, 
judging from the various comments made on the evaluation form as well as 
from those garnered from informal discussions, it would appear that this 
may have been due to the manner in which the material was presented and 
the fact that some presenters were not very dynamic. Next in order of 
"least" liked were the sessions on sampling. Six respondents indicated 
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sampling, all of them from Workshop L Three persons from Workshop 
I felt that "repetition of information already known** was least liked, and 
three indicated that the ^'speakers'* were least liked. Two persons in 
Workshop i wrote in "data collection. " From Workshop III no response 
was greater than one, except "report preparation. " And three participants 
in Workshop III said that everything was of value. Some of the individual 
comments made by the Workshop III participants are: 

"Friday PM session--too much on NAEP problems. " 
"Exercise Development" 
"Utilization /Application" 

"Report writing- -insufficient detail. Maybe optional examples 
should be discussed." 

The participants in Workshop I had more to say on this question 
than their counterparts in Workshop III. The six persons who indicated 
sampling as least liked in Workshop I indicated that it was because of the 
technical nature of the material. These participants expressed a desire 
for a less-technical presentation, and would rather have had a survey of 
sampling presented than a discussion of how a particular sample is selected. 
One reaction was ". . .the sampling discussion (which was too quick, too 
complicated), although it was directed by the most dynamic speaker. " 

Inasmuch as some of the sessions that were least liked were tech- 
nical and some were of a non-technical nature, one might conclude that 
the least liked feelings are related either to specific content or to the pre- 
senters their organization, content, presentation, and the like. Otlier 
individual comments made as "least liked" were as follows: 

"General lecture attitude" 

"Failure to elaborate on points" 

"Time precluded questions and discussion" 

In the main, the majority of participants had more "most" than 
"least" comments about their workshop experience. Three of the 
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participants in Workshop III and two persons in Workshop I said that all 
the sessions were good and valuable. 

The least liked reactions related primarily to the presenters and 
to the technical nature of some of the material. In addition, some of the 
**least*' comments were related to 'Uack of the respondents responsibility 
for or contact with this activity. 

The reactions to the questions about most and least liked portions 
of the program suggest that state department assessment personnel want 
to know more about what^s going on in other assessment programs and 
want to have the opportunity to meet and discuss problems of mutual con- 
cern. In a sense, the workshop serves as a cathartic agent and helps to 
give confidence to those who are working in a very difficult area. 

Degree of Satisfaction with Workshop Experience 

There were three specific questions asked in order to provide 
objective evidence regarding the future of the workshop. One of these 
was ''Would you recommend the workshop for other State Department 
personnel?** Another was "Would you recommend this workshop for local 

education agencies (LEA's)?** A third question was "Would you be interested in 

attending a follow-up workshop next summer?" The responses to these three questions 

Number of Participants Responding 
Workshop I Workshop III 

Question Yes No ? Yes No ? 

Recommendation for other State 

19 4 1 

Department Personnel 
Recommenda^'ion for LEA's 13 10 1 

Attend follow-up workshop 20 1 3 



13 1 1 

2 7 4 
12 2 1 



Over 80 percent of the participants in both workshops (83 percent 
and 86 percent in Workshop I and III) indicated that they would recommend 
this type of workshop to their colleagues in the state department. The 



reasons most often given were: 

"It would provide them with a better understanding of assess- 
ment" (o); 

"These people would benefit greatly from a basic orientation 
to NAEP ana thereby might avoid some of the problems NAEP 
encountered" (5); 

"Need to know what's going on** (3); 
"Exchange of ideas" (2); 

"It would help facilitate communication among staff" (2). 

A comment echoed by many of the participants was "particularly for 
'non-assessment' oriented staff who are important to an overall effort but 
who do not |t now the why or wherefor of assessment (and program plan- 
ning)." AnVher comment made oy some of the participants either explicitly 
on the questioi|naire or in informal discussions was "no matter how much 
we caution pec^Ie about rushing into state assessment without adeqaute 
and careful plai^l^ng, they don't believe us. There are those among us 
who want to collect data tomorrow they need to hear first-hand of the 
problems involved. " 

For the five participants who answered in the negative, the consensus 
was that unless the person is actually involved in a program, he need only 
have a superficial knowledge and understanding which could be obtained 
from existing documents. One participant felt that the material is too 
sophisticated for most state education agency (SEA) personnel and felt that a less 

technical information session for the generalist, curriculum specialists, and department 

managers would be most helpful. 

* * * * 

There was considerable variation between the two groups insofar as 
recommending the workshop to LEA's. Slightly more than one-half (54 
percent) of Workshop I people but only 13 percent of Workshop III partici- 
pants answered in the affirmative. And there were more participants in 
Workshop III (four persons versus one person) who indicated uncertainty. 
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why such a marked difference? The written comments do not contain an 
explanation. Of these who said they would recommend a NAEP workshop 
for LEA*s, the only comment made by more than one person was *lt 
would broaden their perspective on assessment and help to break down 
fear'* (three persons). The reasons given by the participants who said 
**no" to the idea of a NAEP workshop for LEA's said such things as: 

'*The NAEP approach is not applicable to LEA*s'^ 
'*Only good for large LEA's'»; 

''Such sessions should be offered by the SEA to give it a state 
flavor and gear it more to their policy system. " 

Some of those who responded "no'* did comment that there is the 
need for a workshop dealing with basic questions and issues where utility 
should be the first consideration. 

Considering the fact that the workshop was not organized for LEA's 
the response may be more positive than one had a right to expect. In 
fact, had the responses been too positive one might make a case that the 
workshop organized for state department personnel left something to be 
desired. What is evident is that, as presently constituted and conducted, 
this workshop is designed specifically for those persons involved in state 
assessment programs or for the very large LEA's. 

* 'fi ❖ 

Possibly the best indicator of the success of the workshop can be 
found in looking at the number of participants who indicated that they would 
like to return for a follow-up workshop. Collectively, 39 participants 
(82 percent) said they would be interested in attending a follow-up workshop 
next summer; 3 participants (6 percent) said no; and 4 participants (8 
percent) said perhaps. When the results are broken down by workshop, 
the percentage differences are minor, 
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Suggested Iniprovenients 

Although there were open-ended sections following various ques- 
tions, where the respondents could offer suggestions for improving 
future workshops, there were two questions specifically designed to do 
this. They differed slightly in their orientation. One question called 
for improvements for future meetings whereas the other was entitled 
final comments, and the respondents were told to offer suggestions for 
similar and different types of workshops. Since the questions are getting 
at essentially the same thing that is, how to improve on the present 
model the responses will be considered together. Many of the sug- 
gestions were common to the participants, regardless of the workshop 
they attended: (1) select the speakers with greater care (eleven responses); 
(2) avoid evening sessions (two responses); (3) try to have more free 
time built into the program so that people can collect their thoughts and 
converse with others (three responses); (4) try to house people in a central 
location to make interaction more feasible (two responses); (5) keep the 
workshop size small to permit discussion and interaction (two responses); 
and (6) spend more time looking at issues, the "why*s," the relationship 
of the results to the improvement of education, and alternative approaches 
to the NAEP model (three responses). In addition, there were suggestions 
common to both groups that concerned themselves with physical facilities 
of the meeting rooms, coffee breaks, and the like which although valuable 
will not be discussed at this time. These additional points, however, will 
be considered by the co-directors in planning future workshops* 

Three respondents suggested that each speaker prepare a brief out- 
line which is distributed either at the beginning of the workshop or mailed 
to the participants before the workshop. This might eliminate premature 
questions and discussion as well as focus the participant's thoughts, if 
the materials were read prior to attendance. 

Three, or four individuals suggested that the total group be broken 
down into smaller groups, each of which would, in an actual working 
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situation, carry through a simulated assessment program from the 
beginning of identifying objectives to the preparation of a final report. 
Another type of stratification suggested was on the basis of interest 
areas. Still another type of grouping that was suggested was that there 
be a few major topical sessions where all participants would be present 
and then a series of small groups where selected areas would be con- 
sidered in more depth. These comments suggest that some of the partici- 
pants would like to attend a '^working-through simulated set of experiences** 
type workshop rather than having the material presented to them in either 
a lecture or discussion medium. It should be noted that an attempt some- 
thing along these lines was tried in a previous workshop; but the partici- 
pants were not too pleased, feeling that they might have missed something 
by being in one group rather than another. 

Two people asked whether it would be possible to have a 1-1/2 day 
information-type session showing NAEP products and discussing NAEP's 
future plans. Two participants commented upon the marathon-speed in 
which the workshops were conducted, stating that at times the audience 
missed some important material either because of the tempo and/or 
because they were tired. Six participants commended the co-directors 
and staff on their presentation of an excellent workshop. 

One participant said, ^'Though I may have sounded negative at times, 
I felt good about the workshop and I felt it was worthwhile. Other com- 
ments were "the workshop has been worthwhile. . . it was well organized**; 
"the program content was most appropriate**; and **the directors set a 
pleasant atmosphere for an open conference which encouraged discussion. *' 

In summary, it is evident that the majority of the participants in 
both Workshops I and III were satisfied with their experience. 

Conclusions and Recommendations 

On the basis of the workshop evaluation, as well as from informal 
discussions with the participants, it is evident that Workshop I and III 
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were quite successful. It would appear that most of the participants 
were pleased with the overall structure, content, materials, facilities, 
and opportunity for individual interaction. However, about a third of 
Workshop I were dissatisfied with the opportunity given them to engage 
in group discussion, and many felt that the speakers were average. 

Many of the participants stated that as a result of attending the 
workshop, they developed a better understanding of National Assessment 
in general, and their local plans in particular. Many stated that they 
were able to meet their colleagues from other states and discuss simi- 
larities, differences, problems, solutions, and the like to a common 
goal that of undertaking a state assessment. This, of course, was a 
major purpose of the workshop. It was not the purpose of the workshop 
to promote a particular assessment model (the National Assessment 
model). But, it was the intent to give participants a detailed view of 
National Assessment so that they could use the information in developing 
their own assessment plans. In this way, hopefully, future state and 
local assessments can profit by National Assessment experiences, and 
avoid some of the problems experienced by NAEP. As one previous 
workshop participant stated, "there is no reason to re-invent the wheel 
when it may only require some slight modification." 

Based upon the various responses made on both the questionnaire 
and in informal discussion, we offer the following recommendations: 

1. National Assessment should sponsor another workshop for new 
personnel in state departments undertaking or contemplating 
conduct of an assessment program. 

2. Although three-fourths of the respondents indicated satisfaction 
with the amount of time for the workshop (three days) the specific 
suggestions for more discussion (Workshop I), the concern about 
evening sessions, and a potential desire for program differentiation 
suggest that serious consideration should be given to the time 
needed for this workshop. The six-month follow-up survey should 
include questions that seek more information on this point. 
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3. Consideration should be given to reorganizing the overall workshop 
so that all participants need not all attend the same sessions. 
Some participants commented that some of the sessions were too 
elementary or that they were already familiar with the material 

or that they had no interest or responsibility in that area. Pos- 
sibly the general discussions of NAEP methodology should be 
reduced and participants be given more time to pursue their own 
specific interests. 

4. The maximum size for a workshop like these two should be 15-20. 
A large meeting begins to resemble a conference, and lessens 
the opportunities for participant interaction. 

5. If pre-registration indicates an enrol?.ment larger than 20 the 
total group should be broken down according to their familiarity 
with the NAEP model. A person just appointed to head an assess- 
ment program might or might not be very knowledgeable about 
assessment in general and the NAEP model in particular. 

6. More careful attention should be paid to the selection and prepara- 
tion of speakers. There were instances where knowledgeable 
speakers were ineffective **teachers. " Outside consultants should 
be used when it is known that they are stimulating teachers and 
also have the appropriate skills. 

7. An attempt should be made to cater to some of the needs of par- 
ticular states but not at the expense of the overall philosophy in 
which these workshops are conceived and conducted. 

8. The 1974 meeting should place more emphasis on discussing the 
manner in which the data are disseminated and development of 
objectives. 

9. The 1974 meeting should include more "discussion" sessions, 
with time being made available for such sessions. 

10, National Assessment should sponsor a follow-up assessment 

meeting in 1974 for persons who attended these two workshops. 
The majority of the participants in both Workshops I and III 
indicated that they would be interested in returning next summer 
for a follow-up meeting. This relates to a suggestion for the 
continuation of the Workshop II approach (see recommendations 
after the next section). 
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EVALUATION OF WORKSHOP EXPERIENCES 
WORKSHOP II 



Workshop II was designed for persons who were already familiar 
with National Assessment. The program concentrated primarily on state 
assessments. 

While 53 persons attended one or more sessions, only 44 were in 
attendance for the entire session. Of these, 40 responses to the evalua- 
tions were received. In a few instances the total number of responses 
to a particular question will be less than 40 because of failure to respond 
to that question by oae or two persons. 

Reactions to Conference Structure 

The results for the first question are: 

Good Average Poor ? 



1. 


Overall workshop organization 


36 


4 


0 


2. 


Workshop content 


31 


9 


0 


3. 


Speakers 


17 


23 


0 


4. 


Workshop materials 


26 


12 


1 


5. 


Living facilities 


31 


6 


0 


6. 


Opportunity to interact individually with 
other conference participants 


29 


9 


2 


7. 


Availability of NAEP staff 


30 


6 


1 


8. 


Availability of workshop co-directors 


35 


4 


0 



These results showed that about 90 percent of the respondents felt 
that the overall organization was good, as was the availability of the work- 
shop co-directors. About 75 percent rated the following categories **good": 
(1) workshop content; (2) living facilities; (3) opportunity to interact; and 
(4) availability of NAEP staff. Thus, six of the eight questions indicated 
considerable satisfaction with the workshop. 
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Workshop materials were felt to be good by two-thirds of the 
respondents. This is positive, but may suggest that some thought should 
be given to improvement another year. 

The only response that was rated less than good by a majority of 
the respondents was the questior on speakers 58 percent responded 
'•average*'; 42 percent responded ''good. " This clearly needs some 
attention for next year. Some effort needs to be made to improve the 
quality of presenting material. Since not everyone is really accomplished 
at "teaching" this may be difficult to change. And since many of the 
presenters were chosen because of their position in a particular program, 
it may be difficult to change. However, greater efforts could be made to 
encourage speakers to prepare themselves thoroughly, to have handouts 
and/or satisfactory audio-visual materials, etc. 

Some respondents make comments after these questions. Those 
made by more than one respondent are as follows: 

4 Best workshop ever attended 

4 Good state assessment programs 

3 Poor opportunity to interact 

3 All participants should be housed together 

3 Too much scheduled 



Reactions to Specific Session s 

The responses to the next question are given below: 

Good Average Poor ? 

1. (California, Florida, Maryland, Texas) 

1. 1 Purposes, objectives, exerciser 22 12 2 2 

1.2 Sampling & data collection 21 14 12 

1. 3 Analysis and report writing 19 16 12 

1.4 Dissemination 21 16 0 2 
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Good 


Average 


Poor 






NAEP dinner program 


17 


16 


1 


0 


3. 


What's New at NAEP 


6 


15 


17 


0 


4. 


Minnesota assessment 


23 


14 


2 


0 


5. 


New Jersey assessment 


26 


8 


0 


2 


6. 


Michigan assessment 


22 


11 


2 


0 


7. 


Small group discussions 


20 


9 


1 


1 


8. 


Individual consultation in Denver 


2 


0 


0 


0 



From these tabulations one might conclude that the reactions to 
the various presentations of state assessment plans were quite consistent, 
with the fol]owing numbers rating them as **good»» -- 22, 21, 19, 21, 23, 
26, 22 (questions 1, 4, 5, and 6). However, percentages ranged from 50 
to 72. In toto, the three separate presentations (Minnesota, New Jersey, 
and Michigan) were received a bit better than the four "combined*' presen- 
tations. Perhaps of greater significance is the fact that the respondents 
reacted quite differently to different sessions. Only four respondents 
^^^^^ iJi sessions as either good or average; everyone else cast a 
split ballot. This suggests that everyone felt that at least some of the 
state assessment sessions were good. In fact, only two respondents did 
not check at least one session as being good. So, different needs were 
being met. 

The NAEP dinner program received a 50-50 reaction. But four 
respondents either double-checked that program as being outstanding or 
very good or added a footnote to that effect. 

The only generally negative reaction was to the discussion of what's 
new in NAEP. One can only speculate as to whether there is a lack of 
interest in NAEP activities or wliether the things that are new are not of 
great interest or whether too much time was devoted to it. 

The small group discussion sessions were well received. Perhaps 
more time should be devoted to discussion sessions in the future. 

***** 
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Reactions to Length of Workshop 

The question about balance of time and material resulted in 24 
respondents saying that it was satisfactory; 9 saying that material should 
have been spread over more days; and 4 saying that less material should 
be covered in the same number of days. The co-directors had been con- 
cerned about "working" the participants too much. But only about a third 
of the respondents felt that the balance was unsatisfactory. 

Some individual comments were made to that question. Six respon- 
dents suggested eliminating evening sessions; three suggested adding a 
day or two; two suggested that the schedule was OK; and two suggested 
fewer days but the same amount of material. 



Most and Least Liked Sessions 

Two of the questions dealt with those things that were of most value 
and those of least value. There was considerable agreement on each 
question. Eighteen persons specifically mentioned things like "sharing 
ideas with others"; "interacting with others"; or "discussing problems 
of state assessment," as being most liked. Thirteen persons specifically 
mentioned that the most liked reaction was hearing about other state 
assessments. These two reactions were the overwhelming ones. No 
other single reaction was mentioned by more than two respondents each. 
The points that were mentioned by at least two persons were; (1) total 
impact of workshop; (2) sampling; (3) getting help from NAEP; and (4) 
the state plus contractor presentations. 

The least liked program was "What's New in NAEP. " Eighteen 
persons mentioned that one. Of the eighteen, seven pointed out that it 
was repetitious of things covered in WorkshopI which they attended. 
Three other reactions were mentioned as least liked by more than one 
person. They were; (1) technical discussions; (2) too much detail; and 
(3) the dinner meeting. 
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These reactions confirm those from other years that tell us that 
state department assessment personnel want to know more about what 
is going on in other states and v/ant to be able to share and discuss their 
own assessment questions and concerns. 

***** 

Recommendation for Workshop for Others 

Thirty-four of the respondents (9O percent) indicated that they 
would recommend a workshop of this nature for other state department 
personnel. This certainly suggested an overwhelming feeling of satis- 
faction with the total impact. Seventeen of the 34 said that their 
reason was because non-assessment personnel in state departments 
(curriculum specialists were most often mentioned) need to know more 
about assessment. Sixteen of the 34 indicated that they would 
recommend the workshop because they received a great deal from it 
themselves. Thus, respondents really reacted to this question quite 
differently some in relation to its suitability for non-assessors, some 
in relation to its suitability for assessors. In one sense, then, it turned 
out to be a double-barreled question, which is difficult to interpret except 
to conclude considerable satisfaction. 

* %c * * 

When asked if they would recommend this workshop for LEA's, the 
responses were more mixed. ' Twenty-one said no, 13 said yes, and 
five said maybe. Of those who said no the most common reasons were: 
(1) it's the SEA'S job to deal with LEA»s, or (2) it doesn^t apply to LEA»s, 
or (3) too technical. Of these who said yes the most common reasons 
were: (1) LEA's need to know about assessment too, or (2) would recom- 
mend for large cities only. 

*>:<*>;: 
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Suggestions for Improvement 

Two of the questions were completely ope'n-ended. One asked for 
suggestions for improving the meeting; one asked for **final comments, 
most of which were suggestions for improvements. So it seems appro- 
priate to summarize those comments together. Certain points were 
repeated often enough (from three to eight times each) to warrant mentioning 
them. 

One common response was ''to continue with presentations of specific 
state plans** (seven responses). This suggestion reinforces the very- 
positive responses that were received on questions relating to specific 
state assessment plans. 

The small group discussion sessions were asked for again (five 
responses) with several respondents suggesting that more time be allowed 
for each session. The hour allocated was not felt to be sufficient, accord- 
ing to some comments. 

Quite a few respondents (eight) suggested more '^focused" sessions, 
in which a specific topic would be addressed in some depth perhaps a 
half day devoted to it (or more). Specific topic suggestions were "some 
subject area such as reading"; "what's new in various states or NAEP'*; 
"new technology"; "assessing change. " 

Seven respondents suggested that the 50 states were different enough 
on some bases (size, geography, level of assessment "sophistication" or 
group vs. individual pupil data) to ask for breaking up the workshop. One 
type of suggestion was to move to mini-conferences or to regional con- 
ferences. Another type of suggestion was to offer more alternative ses- 
sions within an annual meeting. 

Five respondents suggested the need for better organization of 
presentations and/or for better speakers. It was felt that some presenters 
had done a much better job of preparation than others. 

Three persons called for more attention to the relationship of 

34 



ERLC 



44 



assessment to instruction, of focusing on how assessment results can 
be used more effectively. Several suggested bringing more curriculum 
specialists from their own .staffs to the meeting. 

A great maiiv comments were individual ones. They will not be 
listed here, but \^ili be reviewed carefully in planning for future activities. 
One individual suggestion will be quoted however. It is "Fine each 
speaker 10^^ for each time he says, 'you know. ' The fund would pay 
all refreshment costs." 

'i' '\' ^ 

Best Time for 1974 Meeting 

When asked to suggest the time for a similar meeting in 1974, the 
rej>»ilts were as follows: 



May 5 

June 4 

July 9 

August 4 

October I 

Before June 1 

Summer I 

After July 1 3 

* i;: 



Would You Like to Attend in 1974 ? 

The most rewarding response, from the director^ point of view, 
was made to the last question. ^'Would you be interested in attending a 
follow-up workshop next summer?" Thirty-seven persons said yes, two 
persons said perhaps, and no one said no. This response leads directly 
to our first recommendation, below, 

j's '*r 
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Recommendations Based on the Evaluation 



The directors* evaluation of the workshop evaluations has led us 
to the following conclusions: 

1. National Assessment should sponsor a summer workshop (or con- 
ference) on large-scale assessment for state department personnel 
in 1974. Aside from the quality of the program, the fact that the 
meeting brings together state assessment workers is itself a most 
important contribution to the assessment movement. 

2. The 1974 meeting should place considerable emphasis upon what is 
being done in state assessments. 

3. The 1974 meeting should include more ^'discussion*' sessions, with 
more time available for such sessions. 
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4. Efforts should be made to improve the "quality" of presentations, 
either by judicious selection of presenters or more specific direc- 
tions, to the extent that this can be done. The directors do not 
feel, however, that an important topic, or a given state program 
should be eliminated from consideration because of an inability to 
guarantee a "stimulating" presentation. 

5. Efforts should be made to focus on specific assessment "issues" 
of concern to a number of states, and not confine the efforts to 
things of interest to all states. 

6. Efforts should be made to increase the "individualization" of the 
program by offering more choices rather than concentrating as 
much on total group sessions. This suggests a movement toward 
a "conference" format in contrast to a "workshop" format. 

7. Consideration should be given to inviting a greater variety of state 
department personnel to at least some of the meetings next year, 
in particular, curriculum specialists. A conference format would 
lend itself to that possibility. 

8. The six-month follow-up evaluation of workshop attendees should 
be focused on identifying specific program content that will be of 
maximum utility. Suggestions made in the recent evaluation should 
be used as one basis for developing the questionnaire. 

❖ ❖ 

The directors are pleased with the evaluations of Workshop II, 
both because of indications of satisfaction but also because of the many 
specific suggestions for improvement. 
O 36 
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WORKSHOP STAFF 



DIRECTORS 

Frank B. Womer irvin J. Lohmann 

The University of Michigan Michigan State University 

NAEP STAFF 

Assessment Workshop Coordinator 
Jack Schmidt, State Assessment Consultant 

Abbey, Janet, Secretary, Utilization /Applications 
Brown, Rex, Report Coordinator, Utilization/Applications 
Burton, Nancy, Area Development Monitor, Exercise Development 
Crane, Robert, Report Writer, Utilization/Applications 
Gadway, Charles, Writer Analyst, Research and Analysis 
Hazlett, James, Director NAEP, and Director, Elementary and 

Secondary Education Services, ECS 
Herman, Magdalen, Program Consultant, Utilization/Applications 
Johnson, George, Associate Staff Director, Administration 
Martin, Wayne, Analyst, Research and Analysis 
Mughrabi, Linda, Secretary, Utilization /Applications 
MuUis, Ina, Analyst, Research and Analysis 

Newcomb, Scott, Area Development Monitor, Exercise Development 

Oldefendt, Sue, Analyst, Research and Analysis 

Phillips, Donald, Analyst, Research and Analysis 

Pratt, Ted, Dissemination Coordinator, Utilization/Applications 

Rogers, Todd, Sampling Coordinator, Research and Analysis 

Salazar, Teresa, Program Consultant, Utilization/Applications 

Seaman, Ken, Director, Utilization/ Applications 

Searls, Donald, Director, Research and Analysis 

Sherman, Sue, Analyst, Research and Analysis 

Talle, Roger, Director, Operations 

Trussel, Vicky, Editor, NAEP Newsletter, Utilization/Applications 

Vandermyn, Gaye, NAEP Publiclnformatlon Officer, Communications Department/ECS 

Wilson, Hal, Director, Exercise Development 

Wright, David, Data Coordinator, Research and Analysis 

UU JU 

'I" -r» -r ^fi 

SPEAKERS 

Bettinghaus, Erwin, Professor of Communication, Michigan State University 
Miller, Gerald, Professor of Communication, Michigan State University 
Rubin, Louis, Professor of Communication, University of Illinois 

5f: * * s'l 
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STATE DEPARTMENT PRESENTERS 



Adams, John, Director, Educational Assessment, Minnesota 
Ascher, Gordon, Director, Educational Assessment, New Jersey- 
Carlson, Dale, Director, Statewide Assessment Programs, California 
Cruse, Keith, Program Director, Needs Assessment, Texas 
Fisher, James, Consultant, Division of Instruction, Maryland 
Keiter, Roberta, Assistant Director for Test Administration, Montgomery 

County Public Schools, Rockville, Maryland 
Impara, James, Administrator, Statewide Assessment, Florida 
Schooley, Dan, Director of Test Development, Michigan 
Shepard, Lorrie, Program Evaluation & Research, California 

♦ 4e :!e :te 

ASSESSMENT CONTRACTORS PRESENTERS 

Bayless, David, Senior Educational Sampling Statistician, RTI 
Pyecha, John, Senior Educational System Analyst, RTI 

Finch, Fred, Manager, Custom Instrument Development, CTB/McGraw-Hill 
Schabacker, William, Elementary and Secondary Programs, ETS 

4e :!c sSe :te :9c 

UNITED STATES OFFICE OF EDUCATION 

Majors, Dexter, Program Officer, State Agency Cooperation Division 
Song, Tongsoo, NCES, Office of Education 

sQc ^ iQC 7^ 

COOPERATIVE ACCOUNTABILITY PROJECT 

Ludka, Art, Assistant Director 
Olson, Arthur, Director 

♦ ^jf ♦ !{S jjC 
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WORKSHOP 

ALABAMA 

Russell Chandler 

Research Consultant 

Office of Planning & Evaluation 

412 State Office Building 

Montgomery 36104 

Frank T. Speed, Director 
Departmental Services 
Room 416, State Office Bldg. 
Montgomery 36104 
Phone: 205-269-7216 

ARIZONA 
Eugene Hertzke 

Superintendent, Creighton Distri 
2702 E. Flower Street 
Phoenix 85016 

CALIFORNIA 

Dale Carlson, Director 
Statewide Assessment Programs 
Department of Education 
721 Capitol Mall 
Sacramento 95814 

Lorrie Shepard , Consultant 
Office of Program Evaluation 

and Research 
Department of Education 
721 Capitol Mall 
Sacramento 95814 
Phone: 916-322-2200 

COLORADO 

John F. Haberbosch, 
Senior Consultant, District 
Planning Services 
Department of Education 
State Office Building 
Denver 8020S 

John W. Helper, Consultant 
Department of Education 
Room 430, State Office Bldg. 
Denver 80203 
^ Phone: 303-892-2238 
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PARTICIPANTS 



Jim Hennes 

Planning & Evaluation Unit 
Department of Education 
State Office Building 
Denver 80203 
Phone: 303-892-2217 

CONNECTICUT 

Joseph J. Cashman 

Educational Consultant 

Bureau of Management & Finance 

State Department of Education 

PO Box 2215 

Hartford 06115 

George Kinkade 

State Department of Education 

State Capitol Building 

Hartford 06115 

Phone: 203-566-4382 

FLORIDA 

James C. Impara, Administrator 
Department of Education 
117 M.J. Suite 10 
Tallahassee 32304 

Tom Seals, Evaluation Consultant 
Department of Education 
Suite 10 M.J. 
Tallahassee 32304 

Gambhir M. Shrestha 
Statistician, Evaluation Section 
SUte Department of Education 
Tallahassee 32304 

GEORGIA 

Peter Thomas, Consultant 
Statewide Testing 
State Office Building, Room 258 
Atlanta 30334 
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HAWAII 



MARYLAND 



Janet I. Sumida, Evaluator 
State Department of Education 
1270 Queen Emma Street 
Honolulu 96813 

KENTUCKY 

William W. Field 

Program Coordinator 

LEA Title I Projects 

Division Compensatory Education 

Department of Education 

Frankfort 40601 

David W. Shannon, Head 
Office of Planning & Evaluation 
Department of Education 
Frankfort 40601 

Eugene Tallent, Coordinator 
Planning and Evaluation 
Bureau of Instruction 
Department of Education 
Frankfort 40601 
Phone: 502-564-3233 

Donald S. VanFleet, Director 
Division of Evaluation 
State Department of Education 
Frankfort 40601 
Phone: 502-564-4394 

MAINE 

Joseph Pecoraro 

Educational Planner 

Office of Planning, Evaluation, 

and Research 
State Department of Educational 

and Cultural Services 
Augusta 04330 
Phone: 207-289-2373 
207-239-2321 



Robert W. Armacost, Specialist 
Office of Advanced Planning & 

Development 
Baltimore City Public Schools 
2330 St. Paul Street 
Baltimore 21218 

James L. Fisher 
Department of Education 
PC Box 8717 

Friendship International Airport 
Baltimore 21240 
Phone: 301-796-8300 

M. Roberta Keiter 

Assistant Director, MCPS 

Dept. Pupil & Program Appraisal 

850 North Washington 

Rockville 20850 

Phone: 301-279-3596 

James Morgan 

Montgomery County Public Schools 
850 North Washington 
Rockville 20850 
Phone: 301-279-3595 

MASSACHUSETTS 

James F. Baker 
Associate Commissioner 
Department of Education 
182 Tremont 
Boston 02111 
Phone: 617-727-8477 

Steve Kaagan, Executive Planner 
State Education Department 
178 Tremont 
Boston 02111 
Phone: 617-727-5700 
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MICHIGAN 



NEVADA 



Thomas H. Fisher 

Research Consultant 

Educational Assessment Program 

Department of Education 

PO Box 420' 

Lansing 48902 

Phone: 517-373-1830 

Dan Schooley 

Director of Test Development 
Department of Education 
PO Box 420 
Lansing 48902 
Phone: 517-373-1830 

MINNESOTA 

John W. Adams, Director 
State Educational Assessment 
Department of Education 
731 Capitol Square Building 
St. Paul 55101 

Floyd Keller 

Director of Instruction 

Department of Education 

550 Cedar 

Capitol Square 

St. Paul 55101 

Phone: 612-296-4060 

MISSOURI 

John F. Allan, Chairman 
Statewide Assessment Project 
Director of Planning & Evaluation 
Department of Education 
Jefferson City 65101 
Phone: 314-751-3501 



R, H. Mathers, Consultant 
Department of Education 
Heroes Memorial Building 
Carson City 89701 
Phone: 702-882-7111 

NEW JERSEY 

Gordon Ascher, Director 

Office of Educational Assessment 

Department of Education 

225 W, State Street 

Trenton 08625 

Phone: 609-292-7983 

Joyce Matthews, Program Planner 
Office of Assessment 
Department of Education 
225 W. State Street 
Trenton 08625 
Phone: 609-292-6939 

NEW MEXICO 

Alan D. Morgah, State Director 
Evaluation and Assessment 
Department of Education 
Santa Fe 87501 
Phone: 505-827-2928 

NEW YORK 

Alan Robertson, Director 
Division of Education Evaluation 
Office of Research, Communica- 
tions, & Evaluation 
State Department of Education 
Albany 12224 
Phone: 518-474-3236 
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MONTANA 

J. Michael Pichette 
Reporting Services Coordinator 
Research, Planning, Development, 

and Evaluation 
Office of Superintendent of Public 

Instruction 
Helena 59601 
Phone: 406-449-3693 
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NORTH CAROLINA 



PENNSYLVANIA 



William J. Brown 
Director of Research 
State Department of Public 

Instruction 
Raleigh 27602 

Robert C. Evans, Director 
Assessment and Evaluation Unit 
Division of Research 
State Department of Public 

Instruction 
Raleigh 27611 

OHIO 

John Adams, Chief of Planning 
Division of Planning & Evaluation 
65 South Front Street 
Columbus 43215 

OREGON 

R. B. Clemmer 

Coordinator, Planning & Evaluation 

Department of Education 

943 Lancaster Drive, NE 

Salem 97310 

Phone: 503-378-3074 

Robert H. Mattson 
College of Education 
University of Oregon 
Eugene 97401 
Phone: 503-686-3405 

Charles Nelson 
Evaluation Specialist 
Planning and Evaluation 
Department of Education 
Salem 97310 
Phone: 503-378-3074 
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William W. Burson 
Educational Research Associate 
Division of Educational Quality 

Assessment 
PO Box 911, DOE 
Harrisburg 17109 
Phone: 717-787-4234 

Thomas E. Kendig, Chief 
Division of Educational Quality 

Assessment 
PO Box 911, DOE 
Harrisburg 17126 
Phone: 717-787-4234 

RHODE ISLAND 

Carol A. Kominski 
Statewide Testing Consultant 
Division of Research & Planning 
State Department of Education 
Rogers Williams Building 
Providence 02808 

SOUTH CAROLINA 

W. D. Ellis, Director 
Office of Research 
Department of Education 
Columbia 29201 
Phone: 803-758-2169 

Charles R. Statler 
Head Supervisor 
State Testing Program 
Office of Research 
Department of Education 
Columbia 29201 
Phone: 803-758-2301 

SOUTH DAKOTA 

Henry Kosters, Associate 
Superintendent, Planning Division 
Department of Public Instruction 
Pierre 57501 
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TENNESSEE 



VIRGINIA 



Thomas G. Caulkins 
Director of Evaluation 
State Department of Education 
263 Cordell Hull Building 
Nashville 37219 
Phone: 615-741-3206 

John E. Cox 

Assistant to the Commissioner 
State Department of Education 
142 Cordell Hull Building 
Nashville 37219 
Phone: 615-741-2927 

John N. Hooker, Director 
State Testing Services 
1000 White Avenue 
University of Tennessee 
Knoxville 37916 
Phone: 615-974-5385 

Ross Mol, Director 
Program Planning 
State Testing Services 
1000 White Avenue 
University of Tennessee 
Knoxville 37916 
Phone: 615-974-5385 

TEXAS 

Keith L. Cruse 
Program Director 
Needs Assessment 
Texas Education Agency 
201 East Eleventh Street 
Austin 78701 

Ronald Knight 
Office of Planning 
Texas Education Agency 
201 E. Eleventh Street 
Austin 78701 
Phone: 512-475-2066 
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James M. Bagby 
Supervisor of Testing 
State Department of Education 
Richmond 23216 
Phone: 804-770-2615 

WASHINGTON 

Gordon Ensign, Supervisor 
Program Evaluation 
State Superintendent of Public 

Instruction 
Old Capitol Building 
Olympia 98504 
Phone: 206-753-3449 

Robert C. Munson, Supervisor 
Program Evaluation 
State Superintendent of Public 

Instruction 
Old Capitol Building 
Olympia 98504 
Phone: 206-753-3449 

WEST VIRGINIA 

Philip F. Thornton, Consultant 
Assessment and Evaluation 
Department of Education 
Building 6B, Room 363 
Charleston 25305 
Phone: 304-348-2698 

WISCONSIN 

James H. Gold 
Assessment Coordinator 
Research, Assessment, Evaluation 
Department of Public Instruction 
126 Langdon 
Madison 53702 
Phone: 608-266-7941 

Robert H. Gomoll, Section Chief 
Research, Assessment, Evaluation 
Department of Public Instruction 
126 Langdon 
Madison 53702 
Phone: 608-266-1782 

53 



WYOMING 

Peggy A. Bagby 
Assistant Director, WYENAP 
College of Education 
University of Wyoming 
Laramie 82070 

PUERTO RICO 

Martin Alvarado Martinez 
Department of Education 
HatoRey, Puerto Rico 00919 
Phone: 765-9350 

VIRGIN ISLANDS 

Kurt Komives 
Evaluation Specialist 
ESEA, Title III 
Department of Education 
PO Box 630 
St. Thomas 00801 



John O'Neill, Manager 
Measurement Research Center 
PO Box 30 

Iowa City, Iowa 52240 
Phone: 319-356-3234 

Barbara Sizemore 

Associate Secretary 

American Association of School 

Administrators 
1801 N. Moore 
Arlington, Virginia 22209 
Phone: 703-528-0700 

Lyman J. Smith, Director 
Assessment Operations 
Science Research Associates 
259 East Erie 
Chicago^ Illinois 60611 



OTHER PARTICIPANTS 
Joane Binkley 

ERIC Clearinghouse on Social 

Studies Education 
855 Broadway 
Boulder, Colorado 80302 

Robert Fox 

ERIC Clearninghouse on Social 

Studies Education 
855 Broadway 
Boulder, Colorado 80302 

George Heigho, Project Director 

Guidance Programs Laboratory 

Science Research Associates 

259 East Erie 

Chicago, Illinois 60611 

Phone: 312-944-7552, ext. 187 



Bernard H. McKenna 

Professional Associate 

Instruction & Professional Development 

National Education Association 

1201 Sixteenth, NW, Room 715 

Washington, D.C. 20036 




1973 ASSESSMENT WORKSHOP 
EVALUATION FOR WORKSHOPS I AND III^^ 



June 1973 



Boulder, Colorado 



The purpose of this evaluation is to determine the impact of your experi- 
ence here in relation to your local assessment needs. Because of a 
possible follow-up, we request that you identify yourself so that we can 
collate the results of the two evaluations. All information will be kept 
confidential. Only group results will be reported. Thank you in advance 
for your cooperation. 



1. Name of parti cipantj_ 

2. State: 



3. 



Listed below are the eight major topical areas of this workshop. 
Please rank them in descending order of importance to you in the 
performance of your present and (possible) future work. (Use 
or most important.) 



) Development of Objectives 
) Samplings 
) Scoring 

) Report Preparation 



( ) Exercise Development 

( ) Data Collection 

( ) Data Analysis 

( ) Di s s emina ti on 



4. What are your general impressions of the value to you of the following: 



a. Overall workshop organization 

b. Workshop content 

c. Speakers/discussion leaders 

d. Workshop materials 

e. Living facilities 

f . Opportunity for group discussion 

g. Opportunity to interact individually 
with other conference participants - 

h. Availability of workshop co-directors 



good 
good 
good 
Zood 
good 
good 

good 

ffood 



average 
average 
average 
average 
average 
average 

average 

average 




Comments: 



The workshop this year is very condensed; a great deal of material 
is covered in a very short time, including evenings. Please give us 
your evaluation of this concern. 

( ) The amount of material and period of time was satisfactory. 

( ) The amount of material was OK, but should have been spread 
over more days. 

( ) The number of days was OK, but less material should be covered. 
Comments: 



^^This version is condensed slightly by reducing the space available for 
writing comments. 
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6. Who.t improvements would you recommend for future workshops 
(content, speakers, facilities, organization, length, etc.)? 



ERIC 



7. What was of MOST value to you in this workshop? Why? 



8. What was of LEAST value to you in this workshop? Why? 



9. Would you recommend this workshop for other State Department 
personnel? Yes No Why/Why Not? 



10. Would you recommend this workshop for LEA's? 
Yes No Why/Why Not? 



11. Final Comments (please give us your suggestions for improvements 
or for new or different workshops); 



12. Would you be interested in attending a follow-up workshop on national 
and state assessments next summer? 

Yes No 
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1973 ASSESSMENT WORKSHOP 
EVALUATION FOR WORKSHOP Ip:- 



ERIC 



June 1973 Boulder, Colorado 

The purpose of this evaluation is to determine the impact of your experi- 
ence here in relation to your local assessment needs. Because of a 
possible follow-up, we request that you identify yourself so that we can 
collate the results of the two evaluations* All information will be kept 
confidential. Only group results will be reported. Thank you in advance 
for your cooperation. 

1. Name of participant: 

2. State: 

3. What are your general impressions of the value to you of the following: 

a. Overall workshop organization good ^average poor 

b. Workshop content good average poor 

c. Speakers good average poor 

d. Workshop materials good average poor 

e. Living facilities good ^average poor 

f . Opportunity to interact individually 
with other conference participants 

g. Availability of NAEP staff good average poor 

h. Availability of workshop co-director s good average poor 

Comments: 



good average poor 



4. What are your general impressions of the value to you of the following 
specific sessions: 

4. 1 Four state part discussion (California, Florida, Maryland, Texas) 





a. 


Purposes, objectives, exercises 


good 


average 


poor 




b. 


Sampling and data collection 


good 


average 


poor 




c. 


Analysis and report writing 


good 


average 


poor 




d. 


Disseminati on 


good 


average 


poor 


4. 


2 


National Assessment dinner 
program 


good 


average 


poor 


4. 


3 


What^s New at NAEP 


good 


average 


poor 


4. 


4 


Minnesota assessment 


good 


average 


poor 


4. 


5 


New Jersey assessment 


good 


average 


poor 


4. 


6 


Michigan assessment 


good 


average 


poor 


4. 


7 


Small group discussions 


good 


average 


poor 


4. 


8 


Individual consultation in Denver 


good 


average 


poor 



(omit if not applicable) 



*This version is condensed slightly by reducing the space available for 
writing comments. 
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5. The workshop this year is very condensed; a great deal of material 
irs covered in a very short time, including evenings. Please give us 
your evaluation of this concern, 

( ) The amount of material and period of time was satisfactory, 

( ) The amount of material was OK, but should have been spread 
over more days, 

( ) The number of days was OK, but less material should be covered, 
C omment s : 

6, What improvements would you recommend for future workshops 
(content, speakers, facilities, organization, length, etc. )? 



7, What was of MOST value to you in this workshop? Why? 



8. What was of LEAST value to you in this workshop? Why? 



9. Would you recommend this workshop for other State Department 
personnel? Yes No Why /Why Not? 



10. Would you recommend this workshop for LEA*s? 
^Yes No Why/Why Not? 



11. Final Comments (please give us your suggestions for improvements or 
for new or different workshops): 



12, Would you be interested in attending a follow-up workshop on national 
and state assessments next summer? 

Yes No 
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Education Commission of the States 




The Education Commission of the States is a 
nonprofit organization formed by interstate compact 
in 1966. Forty-seven states and territories are now 
members. Its goal is to further a working relationship 
among state governors, legislators and educators for 
the improvement of education. This report is an 
outcome of one of many Commission undertakings at 
all levels of education. The Commission offices are 
located at 300 Lincoln Tower, 1860 Lincoln Street, 
Denver, Colorado 80203. 
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